Computer-implemented system and method for automated and highly accurate plaque analysis, reporting, and visualization

ABSTRACT

A computer-implemented system and method of intra-oral analysis for measuring plaque removal is disclosed. The system includes hardware for real-time image acquisition and software to store the acquired images on a patient-by-patient basis. The system implements algorithms to segment teeth of interest from surrounding gum, and uses a real-time image-based morphing procedure to automatically overlay a grid onto each segmented tooth. Pattern recognition methods are used to classify plaque from surrounding gum and enamel, while ignoring glare effects due to the reflection of camera light and ambient light from enamel regions. The system integrates these components into a single software suite with an easy-to-use graphical user interface (GUI) that allows users to do an end-to-end run of a patient record, including tooth segmentation of all teeth, grid morphing of each segmented tooth, and plaque classification of each tooth image.

The invention described herein was made in the performance of work undera NASA contract, and is subject to the provisions of Public Law 96-517(35 U.S.C. § 202) in which the Contractor has elected to retain title.

BACKGROUND OF THE INVENTION

A. Field of the Invention

The present invention relates generally to plaque removal and whiteningcapabilities for toothbrushes and toothpastes, and, more particularly toa computer-implemented system and method for automatically evaluatingthe efficacy of plaque removal and whitening capabilities for differenttoothbrushes and toothpastes.

B. Description of the Related Art

Gum (or periodontal) disease, including dental plaque, is problematic inAmerican and European households. Almost seventy-five percent ofAmericans and Europeans suffer from gum disease and plaque to someextent. While removal of plaque using dental cleaning devices is aneffective method for plaque control, such removal techniques requirefrequent visits to the dentists which are time-consuming and expensive.Brushing one's teeth is one of the most economical and time-effectivemethod of plaque control. However, negligible work has been done on theanalysis of the efficaciousness of tooth brushing methods to maintaingingival health. While the toothbrush features that control plaqueremoval, such as, e.g., handle size, head size, bristle configurations,bristle patterns, etc. are well-known, no work has been done to date onthe efficacy of brush design for plaque removal on a tooth-by-toothbasis.

Recently, a system to determine plaque removal efficacy was implemented.The system measured the plaque on teeth as disclosed by the fluorescenceof the teeth under ultraviolet (“UV”) light. UV light makes plaque onteeth fluoresce as a yellow color, enamel as a light blue color, gum asa black color, and plaque on gum as a green color. In the system a setof each patient's front teeth was visually and manually overlaid with asynthetic template set for alignment, and then a digital image of themanually-aligned teeth was obtained. A simple Mahalanobis-distance basedclassifier was used to classify each pixel as plaque or enamel. TheMahalanobis distance is a very useful algorithm for determining thesimilarity of a set of values from an unknown sample to a set of valuesmeasured from a collection of known samples. The ratio of plaque versusenamel yielded a measure of percentage plaque on each patient for allthe teeth combined. Resultant analyses before and after brushing alloweda measure of the efficacy of plaque removal for each toothbrush.

This system suffers from several drawbacks. First, it does not measureplaque removal efficacy for each tooth, and does not consider plaquemeasurements from teeth inside the mouth or on the inner (i.e., lingual)tooth surface. This is a significant disadvantage since sometoothbrushes could remove plaque from teeth at the front of the mouth,but not remove plaque from the teeth in the interior of the mouth due toinsufficient reach. Second, the alignment procedure of the system isvisually performed and inconsistent since individual teeth vary widelyfrom the synthetic set used in the system.

Recently, a robot-based brushing system was introduced to test plaqueremoval on synthetic teeth (“typodonts”). Synthetic plaque was coatedonto the typodonts and different brush heads were used to brush thesynthetic teeth using the robot-based brushing system. A simple imageprocessing system was used to measure the plaque remaining afterbrushing. The image processing task was simple because the plaque onteeth could be easily distinguished from enamel and gum using the RGBvalues. Though this system is consistent since the brushing action doesnot vary much, the use of synthetic plaque is not realistic and does notreflect the presence of plaque on real teeth accurately. Nor does thesystem accurately reflect brushing actions in the mouth where thehard-to-reach teeth typically get brushed less than the front teeth.

Thus there is a need in the art for an automated system for analyzingplaque and whitening on a tooth-by-tooth basis in real-time to assist inthe determination of the usefulness of new toothbrushes and toothpastes.

SUMMARY OF THE INVENTION

The present invention satisfies this need by providing acomputer-implemented system and method for automatically evaluating theefficacy of plaque removal and whitening capabilities for differenttoothbrushes and toothpastes. The system and associated algorithms allowplaque analysis on a tooth-by-tooth basis in real-time, therebyfacilitating clinical evaluation of the effectiveness of different modeltoothbrush designs for plaque-removal efficacy for each individual toothand sub-regions within each tooth, prior to mass-manufacture of thetoothbrush.

Additional advantages of the invention will be set forth in part in thedescription which follows, and in part will be learned from thedescription, or may be learned by practice of the invention. Theadvantages of the invention will be realized and attained by means ofthe elements and combinations, and equivalents thereof, particularlypointed out in the appended claims. It is to be understood that both theforegoing general description and the following detailed description areexemplary and explanatory only and are not restrictive of the invention,as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate one (several) embodiment(s) ofthe invention and together with the description, serve to explain theprinciples of the invention. In the drawings:

FIG. 1 is a schematic diagram showing a computing entity useful in thesystem and method of the present invention;

FIGS. 2A-2C are screenshots showing some of the software features of thesystem and method of the present invention;

FIGS. 3A-3D are graphs providing an intuitive explanation of theconventional gray watershed algorithm;

FIG. 4A is an image showing two touching agricultural products;

FIG. 4B is an image showing a distance transform image of FIG. 4A;

FIG. 4C is a graph showing the basins produced by the cross section ofthe agricultural products shown in FIG. 4A;

FIG. 4D is a graph showing the complement of the cross-sectional graphshown in FIG. 4C;

FIG. 5A is an image showing two touching objects having a largegray-scale variation;

FIG. 5B is a graph showing the basins produced by the cross section ofthe objects shown in FIG. 5A;

FIG. 5C is a graph showing the complement of the cross-sectional graphshown in FIG. 5B;

FIG. 5D is an image of teeth having low contrast boundaries;

FIG. 6A is an image of a high-contrast tooth where the enamel-gumboundary is clearly visible;

FIG. 6B is a Gabor edge-normalized output image of the image shown inFIG. 6A;

FIG. 6C is a tooth image with low-contrast between the enamel-gumboundary;

FIG. 6D is a Gabor edge-normalized output image of the image shown inFIG. 6C;

FIG. 7A is an image of a tooth under examination by the system andmethod of the present invention;

FIG. 7B is an image of the brightest regions of the tooth image shown inFIG. 7A;

FIG. 7C is an image showing the central bright or seed region of thetooth image shown in FIG. 7A;

FIG. 8A is an image showing how the central seed region of FIG. 7C isused on an edge map of the image shown in FIG. 7A;

FIG. 8B is an image showing how the central seed region is grown toyield the segmented image;

FIG. 8C is an image showing how the central seed region is grown toyield the central segmented region;

FIG. 9A is an image showing the seed background region of image shown inFIG. 8C;

FIG. 9B is an image showing how the seed background and foregroundregions are grown to yield the final segmented region;

FIG. 9C is an image showing how the final segmented region of FIG. 9B issuperimposed upon the color image of the tooth;

FIG. 10A shows a grid provided on template synthetic tooth;

FIGS. 10B-10D are images showing the various shapes and sizes of realteeth;

FIGS. 11A-11D are examples of two teeth images where the grids areautomatically overlaid onto the segmented teeth using the template gridof FIG. 10A;

FIGS. 12A and 12B are images showing a comparison between the dualwatershed segmentation algorithm of the present invention (FIG. 12A) andthe conventional watershed segmentation algorithm (FIG. 12B);

FIGS. 13A and 13B are images showing examples of undersegmentation andoversegmentation, respectively;

FIGS. 14A-14D are images showing the grid morphing results (FIGS. 14Band 14D) for tooth images (FIGS. 14A and 14C), respectively; and

FIGS. 15A-15D are examples of a classified labeled output image. greenindicates enamel, and black indicates glare.

DESCRIPTION OF THE EMBODIMENTS OF THE INVENTION

Reference will now be made in detail to the present embodiments of theinvention, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts.

The present invention is broadly drawn to a computer-implemented systemand method for automatically evaluating the efficacy of plaque removaland whitening capabilities for different toothbrushes and toothpastes.The system and method allow automated plaque analysis for every tooth ina patient for both, lingual and buccal surfaces. The system and methodprovide swift capturing and storing of each tooth image, and automaticindexing of each tooth image for retrieval in the future. The systemutilizes segmentation algorithms for segmenting (i.e., separating) eachtooth from surrounding teeth, and classification utilities forclassifying plaque regions.

The system and method allow automated division of a tooth into severalregions called grids. Plaque measurements can be made on each grid.Advanced computer graphics morphing procedures are used to graphicallymorph a grid onto a real tooth based on an ideal grid drawn on asynthetic tooth. The entire set of processing operations (includingsegmentation, grid-morphing, classification, and plaque analyses fordifferent teeth and grid combinations) for teeth in a patient(sixty-four teeth including buccal and lingual surfaces) is typicallyaccomplished in three to six minutes, well within the goals of clinicaltrials for brush evaluation. Another distinct advantage of the systemand method of the present invention is that the images and associatedresults can be retrieved at any time, thereby allowing comparisons ofdifferent studies and brushes taken at different times.

In accordance with the invention and as shown in FIG. 1, the system ofthe present invention includes a conventional computing entity 100, or aseries of connected conventional computing entities. Each computingentity 100 may include a bus 102 interconnecting a processor 104 aread-only memory (ROM) 106, a main memory 108, a storage device 110, aninput device 112, an output device 114, and a communication interface116. Bus 102 is a network topology or circuit arrangement in which alldevices are attached to a line directly and all signals pass througheach of the devices. Each device has a unique identity and can recognizethose signals intended for it. Processor 104 includes the logiccircuitry that responds to and processes the basic instructions thatdrive the computer. ROM 106 includes a static memory that storesinstructions and date used by processor 104.

Computer storage is the holding of data in an electromagnetic form foraccess by a computer processor. Main memory 108, which may be a RAM oranother type of dynamic memory, makes up the primary storage of thecomputer. Secondary storage of the computer may comprise storage device110, such as hard disks, tapes, diskettes, Zip drives, RAID systems,holographic storage, optical storage, CD-ROMs, magnetic tapes, and otherexternal devices and their corresponding drives.

Input device 112 may include a keyboard, mouse, pointing device, sounddevice (e.g. a microphone, etc.), biometric device, or any other deviceproviding input to the computer. Output device 414 may comprise adisplay, a printer, a sound device (e.g. a speaker, etc.), or otherdevice providing output to the computer. Communication interface 116 mayinclude network connections, modems, or other devices used forcommunications 118 with other computer systems or devices.

As will be described below, computing entity 100 consistent with thepresent invention may make be part of the system of the presentinvention and perform the method of the present invention toautomatically evaluate the efficacy of plaque removal and whiteningcapabilities for different toothbrushes and toothpastes in accordancewith the present invention. Computing entity 100 performs this task inresponse to processor 104 executing sequences of instructions containedin a computer-readable medium, such as main memory 108. Acomputer-readable medium may include one or more memory devices and/orcarrier waves.

Execution of the sequences of instructions contained in main memory 108causes processor 104 to perform processes that will be described later.Alternatively, hardwired circuitry may be used in place of or incombination with software instructions to implement processes consistentwith the present invention. Thus, the present invention is not limitedto any specific combination of hardware circuitry and software.

The system and method of the present invention permit swift capture ofclinical image data for plaque and gingival analysis, automatedprocessing and generation of plaque classification results for eachpatient and across different patients in a study. The entire set ofoperations is wrapped in an easy-to-use graphical user interface (GUI)that keeps the details of the implementation hidden from users, andallows people not conversant with computers to easily use the systemwith a minimal learning curve.

Among the features of the software that allows easy use of the systemand method are utilities to create brushes, patients, clinicians, etc.on the fly and assign them to any arbitrary study dynamically (as shownin FIG. 2A), capture images swiftly in any sequence (as shown in FIG.2B), and review the results on all teeth images simultaneously in amosaic dialog (as shown in FIG. 2C) upon completion of processing.

The automated and accurate processing of the captured images to detectplaque on a tooth-by-tooth basis enable performance of clinical trialsand analysis with minimal human intervention. These features offerdistinct advantages over the related art systems. The system and methodinclude an image segmentation technique that robustly segments an objectfrom surrounding background (e.g., gums) and adjacent overlappingobjects (e.g., teeth). A graphics morphing procedure automaticallydivides any arbitrary tooth into several regions (grids) using aprocedure that is robust to teeth orientation changes. This is anontrivial task due to the large variations in teeth shapes, sizes, andorientation within the mouth. Each of the main components of the systemand method of the presented invention is discussed in turn.

A. Object Segmentation

Object segmentation involves the automated determination of objectregions of interest in the presence of surrounding background clutter,noise, and other objects. Numerous object segmentation methods have beendesigned for various applications.

The segmentation technique used for a specific application depends onthe constraints of the problem being solved, including image, color,shape, and texture variability. Stochastic and statistical methods arecommonly used for image segmentation of natural scenes where the objectstructure is not fixed or deterministic. Non-causal Markov models andcausal Markov models have been used for supervised and unsupervisedsegmentation of textured images of natural scenes. These stochasticmethods involve initial pixel labeling using the Markov model parametersor texture features and gray-intensity information, followed byiterative maximum-a-posteriori (“MAP”) re-labeling of the segmenteddata. In addition, autoregressive models, which in essence are GaussianMarkov random fields, have also been used for texture segmentation. Suchtechnologies are powerful when images are characterized by texturedobjects and do not contain definite structure. However, such techniquesinvolve iterative model parameter estimation and stochastic relaxationfor region labeling which makes the segmentation process slow andcomputationally intensive.

Statistical color-histogram approaches to image segmentation includemultilevel thresholding using a histogram with peak-finding followed byregion merging, and region-adjacency graphs to merge adjacent regionswhile considering color statistical information. These techniques do notwork well with teeth due to the close color similarities between plaqueand gum regions in tooth images.

Geometrical edge-based contour models have been used for objectdetection and segmentation. One of the most recent popular approachesincludes active contour models or snakes where an initial curve isdefined and evolved based on true image gradient/edge information orlevel set functions that describe regional texture information.Geometrical models, called active shape models, and variations have beenproposed where the object boundary is described as a series of linesegments for all training images. The mean and the principal componentsof these line segments from training data are used to describe andconstrain the variability in each line segment. For an unknown image,the line positions are moved within the constraints of the trainingsample principal components. Active contour and active shape modeltechniques are sensitive to the initial placement of the contour. Theinitial position needs to be close to the true position. However, thisis not possible for teeth where the sizes, orientation and positionswidely vary.

One of the most robust and quick object segmentation schemes involves awatershed algorithm. A philosophically similar edge-based segmentationmethod utilizes a predictive model to identify the direction of changein color and texture at each image pixel, thereby creating an edge flowvector. The edge flow vectors are propagated to locate the objectboundaries. Initial comparison of the standard gray-watershed algorithmon edge images with the edge-flow technique yields similar segmentationresults on teeth images. However, the edge-flow technique performssignificantly slower (more than ten to one-hundred times slower) andtherefore is not practical for a real-time application wheresegmentation speed is an important consideration.

The system and method of the present invention utilizes a novel,modified watershed segmentation algorithm that yields superior objectsegmentation compared to the standard watershed method. The standardgray and binary watershed algorithms are discussed below, followed by adiscussion of the modified watershed algorithm of the present invention.

1. Gray Watershed Algorithm

This standard algorithm is useful for segmenting an object fromsurrounding clutter. It is applied to gray-scale input images. FIGS.3A-3D depict the steps in the algorithm applied to a one-dimensionalgray-scale profile. In the standard gray watershed algorithm, the centerof each object is assumed to have the highest pixel gray-scale value,with lower values at the object edges. This gray-scale two-dimensionalimage/one-dimensional profile is then complemented so that objectcenters have the lowest values. FIG. 3A shows a hypotheticalcomplemented one-dimensional scan line of a gray-scale image of acluster of two touching objects. It has basins (minima) at the center ofeach object. The algorithm is best described by envisioning filling thiscontour with water, where FIGS. 3B-3D show different water levels in thebasins. This results in the different basins being filled to equalheights. When the water overflows between two adjacent basins, thecorresponding ridges or watersheds locate the dividing lines betweenbasins, or in the current problem, the boundary between adjacent nuts(each nut corresponds to a separate basin).

2. Binary Watershed Algorithm

The binary watershed algorithm is another segmentation algorithm similarto the gray watershed algorithm, and has many variations. This algorithmworks well when objects have a prominent boundary from surroundingboundaries and objects touch/overlap with each other to form clusters.It can be described in terms of erosions, but it is best implementedusing the distance transform. In this case, the object cluster image isbinarized and a distance transform of the binary object cluster iscalculated. The gray watershed algorithm is applied to the gray-scaledistance-transformed image. This combination is referred to as thebinary watershed algorithm. To produce the distance-transformed image,an image is thresholded to obtain a binary blob. Each “on” pixel on anobject is replaced by a gray-scale value equal to its distance from thenearest edge in the object cluster boundary. FIG. 4B shows an example ofthe distance transform image of FIG. 4A. Its cross-section (FIG. 4C)shows two well-defined peaks and is much better than the cross-sectionproduced from the original gray-scale image. FIG. 4D shows thecomplement of the cross-sectional plot in FIG. 4C. Note the prominentvalleys (centers of the nuts) and the peak (the dividing line betweenthe nuts). The gray watershed transformation of FIG. 4D yields the finalsegmented image.

The gray watershed algorithm works well if the objects to be segmentedhave slowly varying continuous gray levels. The algorithm fails onobjects having large internal gray-scale variations, resulting inoversegmentation (where an object is subdivided into several regions),or if the boundary between the object and surrounding background is notcharacterized by a sufficient gray-scale change. FIG. 5A shows anexample of touching objects where the large gray-scale variation in eachobject is shown as a one-dimensional profile in FIGS. 5B and 5C. Thegaps by the shells in these objects can cause significant problems sincethey occur as local minima. The standard gray-scale watershed algorithmlabels prominent local minima as object boundaries thereby resulting inoversegmentation. FIG. 5D shows a typical tooth image where the centraltooth is surrounded by other teeth and gum regions (background). Thecontrast between the tooth and gum regions is hardly noticeable.Additionally, the presence of glare regions caused by reflections due tothe camera light causes high color intensity changes within the tooth.Therefore, the standard gray watershed algorithm does not work well onthese teeth images. Smoothing the image with a Gaussian filter to reducethe effect of local peaks and valleys (to produce smooth images of eachobject) was considered, but this also further reduced contrast betweentooth boundaries.

The standard binary watershed algorithm has over-segmentation problemswhen the object boundary is irregular or complex. Extra segmentedregions and boundary lines were also formed when the objects were notoval or elliptical (this occurs for teeth). In addition, the binarywatershed works well for segmenting two or more touching objects whenthe boundary between object and background is well-defined. In teeth,the boundary between teeth and gum is often ill-defined as shown in FIG.5D. Thus, the standard binary watershed algorithm does not work well onsegmenting teeth.

3. Modified Watershed Segmentation

The system and method of the present invention utilizes a two-stagewatershed algorithm that robustly segments low-contrast objects whilehandling color/gray-scale variations within the object. Prior tosegmentation, the data is preprocessed to handle variations in lightingconditions and variations in object color and contrast. Then a two-stagewatershed-based segmentation is performed, followed by post-processing.

a. Data Processing

Images of objects often have variations in dynamic range and color fromimage to image due to differences in illumination intensity (caused bydifferent light bulb sources), variations in distance between objectsand a light source (intensity is inversely proportional to distancebetween the light and the object), differences in camera hardware (e.g.,hue, saturation setting of the CCD camera, etc.), and variations in thecontrast between the object and the background. Data processing istherefore needed for proper segmentation and classification.

Since there is significant variation in teeth color due to theabove-mentioned reasons, a normalized edge contrast map is extracted tohandle differences in illumination conditions and camera settings. Theinput color image is first filtered using edge kernels in each colorchannel. Gabor edge filter kernels may be used since their scale can beeasily changed mathematically and they have the best jointspace-frequency resolution among all filters. Imaginary Gabor edgefilters have been used extensively in prior work for target recognition,image compression, and recognition of handwritten addresses onenvelopes, among other applications. The Gabor filter outputs from eachcolor channel are weighted and added, to yield a single gray-scale edge(contrast) image. This handles variations in average color betweenobjects and background in different images caused by changes inillumination intensity and/or camera settings.

To handle contrast variability, each edge image is normalized based onthe variance of the edge image. The variance of each edge image iscomputed and each pixel edge value is divided by the image variance.This yields an output edge image with unit variance. This image is thennormalized to ensure that pixels with excessively high values do notskew the image data. This normalization ensures that images withdifferent contrasts have similar normalization output values. FIG. 6Ashows a high-contrast tooth where the enamel-gum boundary is clearlyvisible. FIG. 6B shows the Gabor edge-normalization output image. FIG.6C shows a tooth image with low-contrast between enamel and gum and FIG.6D shows the corresponding output image having good normalizationcontrast. The white spots in both edge images are glare regions that aremasked prior to segmentation.

b. Object Segmentation Algorithm

The two-step watershed algorithm for segmenting objects of the presentinvention is robust to images with low object-boundary contrast andlarge internal color/gray-scale variations. This involves determinationof the central portion of the object with one watershed using the centerof the object as the seed point. This is followed by a reverse watershedusing the external boundary surrounding the object and the internalportion of the object as two seed regions. The points at which these tworegions meet correspond to the true object boundary.

The first stage of the segmentation algorithm involves locating a fewpixels on the object of interest. This can be resolved by one of severaltechniques, depending upon prior knowledge of the specific applicationbeing addressed. The tooth enamel, which is a part of the tooth ofinterest, is typically close to the center of the image and is thebrightest portion of the image due to the incident light and theirnatural color. Therefore, a cumulative histogram-based method is used tolocate a few pixels in the object of interest. Since the dye usedassigns a red color to plaque on teeth and gum regions, the red channelis not useful for discriminating between tooth and boundary regions.However, class-specific useful information is present in the green andblue channels. The cumulative histogram for each of the other colorchannels, green (CHG) and blue (CHB) is determined. The color values (TGand TB) that correspond to the brightest 12.5 percent (%) of the pixelsin each color channel are then determined. Two binary images (BG and BB)are formed by thresholding each of the color channels at the thresholds(TG and TB) and these binary images are combined to yield a singlebinary image, where the brightest enamel regions in the image arelabeled as one (1). This corresponds to finding the brightest 12.5% ofthe regions in both the blue and green channels combined. The algorithmthen labels each connected “on” (white) region in the binary thresholdedimage with a unique color—this process is called “blob-coloring.” Theblob-colored (labeled) image is then analyzed and the area of each blobregion is computed. The algorithm then extracts the largest blob greaterthan an area (AE) closest to the image center. This corresponds to thecentral portion of the tooth of interest in the image. This seed region(SR1) is labeled as one (1) and the rest of the image pixels areassigned to zero (0). FIGS. 7A-7C show an example of the technique todetermine the starting seed region for segmentation.

The seed region is then used as a starting point on the watershedalgorithm applied to the edge map of the color image, as describedearlier in Section A.3.a. The watershed algorithm threshold (TW1) usedin the first stage is very low to allow for detection of a tooth-gumboundary when the image contrast is small. This is the maximum allowablechange in edge value (or maximum allowable edge peak value) in the Gaboredge-map to form a new region in the segmentation procedure duringregion growing. This low value for TW1 ensures that objectundersegmentation does not occur at this stage. Undersegmentation occurswhen a true boundary between an object and boundary is not detected,thereby resulting in fewer segmented regions in the image that areactually present. However, object oversegmentation could occur if a lowthreshold is used. This occurs when a single object is segmented intotwo or more regions. This could occur because of large color/gray-scalevariations within the object (e.g., due to the presence of plaque aroundthe center of the tooth, glare, enamel surface changes, or otherillumination effects). FIGS. 8A-8C show an example of the watershedsegmentation of the image in FIG. 7A using the seed region in FIG. 7C asthe starting point. The edge map in FIG. 8A is used as the input image.Note the oversegmentation result in FIG. 8B. The color variation due tothe plaque region close to the tooth center and the plaque close to thegum line produce spurious segmented regions cause this oversegmentation.The region that is connected to the set of original seed points isselected as the segmented object (tooth) of interest, as shown in FIG.8C. This central segmented region is referred to as S1.

The first stage of segmentation typically yields an oversegmentedobject, as shown in FIGS. 8A-8C. To accurately locate the true objectboundary, the system and method of the present invention implements anew reverse watershed algorithm where the background regions and theobject region that meet during the process of growing each region arelabeled as the true object boundaries. If the object region was grownalone, parts of the background could be labeled as the object if thetrue object-background had low contrast. If too much color variationoccurs within the object, object oversegmentation will occur. Growingthe background region alone is not a good alternative since parts of theobject could be labeled as background if the true object-backgroundboundary has low contrast. Additionally, parts of the background couldbe labeled as object (resulting in undersegmentation) if the backgroundhas too many color variations, due to the presence of other objects suchas surrounding teeth around the central tooth. To avoid all thesedrawbacks, the object and background regions are grown simultaneously,which makes the segmentation process robust to low object-backgroundcontrast, while handling color variations within the object (tooth). Themaximum allowable color variation (the watershed threshold) TW2 isspecified high to avoid object oversegmentation. The surroundingbackground seed region can be chosen using prior knowledge about theapplication. A border region at a certain radius R outside the objectregion from stage 1, S1, is used as the background seed region.

FIGS. 9A-9C show the seed background region chosen with R=70 pixels awayfrom the object seed border. The regions are grown simultaneously usingthe watershed algorithm. The final segmented regions are shown withdifferent labels in FIG. 9B, and the object boundary superposed on thetooth image in FIG. 9C illustrates the accuracy of the segmentationalgorithm. The plaque regions on the tooth around the gum border (bottomportion) and on the biting surface of the tooth (top portion) and thecentral part of the tooth that resulted in spurious regions in the firststage of segmentation (FIG. 8B) are labeled as part of the tooth in thisstage.

B. Object Grid Morphing

After locating the tooth of interest, the tooth has to be divided into anumber (e.g., eight) of regions (excluding the biting surface). Thisdivision of the tooth into eight regions or grids yields accurateinformation about the presence or absence of plaque on different partsof the tooth. This detailed information can be used to predict theefficacy of plaque removal on different parts of each tooth usingvarious toothbrushes. For example, certain toothbrushes could beefficient at removing plaque from the biting surface of a tooth, but notfrom the gingival margin. A grid on a template synthetic tooth is shownin FIG. 10A. There is no fixed rule for overlaying a grid onto areal-tooth. Assigning an area-based rule, or a position-based rule tooverlay a grid on an arbitrary tooth is a difficult task due to the widevariation in shape, size, orientation, asymmetry, and irregularitiesbetween different teeth that is further complicated by the variation indistance between the camera and the tooth. Examples of the variety oftooth shapes, sizes, asymmetry, etc. are shown in FIGS. 10B-10D.

Due to this inconsistency in being able to develop a specific rule foroverlaying a grid onto such a wide variety of shapes, the system andmethod of the present invention employ a computer-graphics basedmorphing procedure to draw a grid on any arbitrary tooth shape.

Several morphing procedures have been implemented in prior work. Themost relevant techniques include mesh-based warping by distortions ofregular grids and extensions to these using deformable snakes. Thesemesh-based methods have problems such as requiring the same number ofpolygons in each model, or when topologies of each object differ (suchas the presence of a hole).

One of the most flexible and effective morphing procedures involvesdescribing the objects of interest using lines. These lines are thesalient features that describe the object. For example, the lines thatdescribe a human face would be those along the eyes, mouth, nose, chin,ears, hairline, etc. When it is desired to morph one object into anotherobject, the line correspondences between each object is firstdetermined. This is typically done by hand in prior work since findingthe line correspondences between the two objects is critical toobtaining good morphing results. In order to warp from one source imageto a destination image, the goal is to find the pixel correspondencebetween the source and destination images. A pixel in the source imageis a function of the pixel location in the destination image and aweighted distance to each line describing the object.

The present invention employs this line-based morphing procedure tooverlay a grid automatically on each segmented tooth. One challenge inthe warping procedure is building the correspondence between lines inthe source and destination images. These lines are typically hand-drawnand the line correspondences are manually selected. When the object sizebetween the source and destination objects differs greatly, and when theshapes are not similar, the correspondence problem magnifies further. Inthe present invention, the source image is the template grid shown inFIG. 10A, and the destination image corresponds to a segmented tooth.The procedure defines an efficient method to describe a segmented toothboundary through lines, and automatically builds line correspondencesbetween the template and a tooth image.

Given a binary segmented object, the center of gravity (C_(x), C_(y)) ofthe binary blob is first determined. The topmost X coordinate of theblob that lies above the center of gravity is noted as (C_(Tx), C_(y)).The number of lines to represent the boundary is defined as N_(B). Thepoints of a boundary representing a segmented object are listed as achain code (an order representation of the boundary points in clockwisefashion), where the boundary starts at (C_(Tx), C_(y)) and proceeds in aclockwise manner. This set of points is then subsampled using digitalFourier transforms to yield N_(B-1) points that represent the boundaryof that object. This subsampling results in N_(B) lines regardless ofthe size of the object boundary, where the starting point is the topmostpoint in the segmented object above its center of gravity. Thesubsampling is also robust to “noise” in the segmented image boundarycaused by artifacts of the segmentation process. This linerepresentation using N_(B) lines is used for both, the template toothand any segmented object blob. This yields a line representation of thetemplate and an arbitrary object with an automatic correspondencebetween each of them. The line-based warping rule is then applied tothese images with the template grid (FIG. 10A) as the source image. Thewarping results in a grid being warped onto the destination segmentimage. FIGS. 11A-11D show examples of two teeth images where the gridsare automatically overlaid onto the segmented teeth using the templategrid in FIG. 10A. The morphing results are excellent even when theshapes of the tooth differ significantly from that of the templatesynthetic tooth image.

C. Plaque Classification

The last step in the system and method of the present invention involvesclassification of plaque regions on the tooth images. The segmentedteeth primarily consist of three types of regions: enamel, glare, andplaque. The classification of glare regions is simple due to theirunusually bright color values. Enamel tends to be similar to plaque inmany cases, so classification between plaque and enamel is often moredifficult. A three-class Fisher Linear Discriminant (FLD) which is aGaussian-based linear pattern recognition classifier is used for thisclassification. The mean and covariance matrices of each class areestimated using the red, green, and blue (RGB) values for training imageregions as input.

D. EXAMPLES

The system and method of the present invention will be further clarifiedby the following examples, which are intended to be purely exemplary ofthe invention. The results of all the significant components of thesystem and method are discussed herein. For all algorithms, a trainingset of sixty-four images from a randomly selected patient record (e.g.,patient A) was used. The training set was used to determine thesegmentation algorithm parameters and the plaque classification matricesfor the three classes. An independent test set of sixty-four images ofanother randomly selected patient (e.g., patient B) was used as the testset to test the performance of the algorithms.

1. Object Segmentation

The segmentation method of the present invention performed significantlybetter than the normal watershed segmentation on the edge image. Inorder compare the performance of the modified watershed segmentation ofthe present invention versus the original watershed algorithm, a subsetof ten typical teeth images was selected from the test patientexamination record, out of the sixty-four test set images. The teethsegmentation parameters were selected based on the performance of thesegmentation algorithm on a set of sixty-four training set images ofpatient A.

The two types of errors that can occur during segmentation areundersegmentation and oversegmentation. Oversegmentation occurs when theobject of interest is divided into several regions after completion ofsegmentation. This implies that only a part of the object is correctlydetected if oversegmentation occurs. An E % oversegmentation erroroccurs when E % of the tooth is not detected after segmentation.Undersegmentation occurs when the segmentation method labels parts ofthe background as belonging to the object. Therefore, the resultantregion will contain other objects besides the object of interest. An E %undersegmentation error occurs when E % of the segmented region containspixels that do not actually belong to the tooth of interest.

The segmentation results of the segmentation algorithm of the presentinvention and the standard watershed algorithm are shown in Table 1below. The dual watershed segmentation algorithm of the presentinvention outperforms the standard watershed on the average segmentationerror per image (4.6% average error versus 16.7% average error). Acertain amount of segmentation error of about 10% is expected andtolerable for this application. As many as six tooth images (out of 10)have greater than 15% segmentation error when the standard watershedalgorithm is used, whereas the dual watershed does not have any imageswith a more than 15% segmentation error.

TABLE 1 Average total No. of teeth (out of No. of teeth (out ofsegmentation 10) with >5% 10) with >15% error (%) segmentation errorsegmentation error Dual 4.6% 2 0 watershed Standard 16.7% 9 6 watershed

FIGS. 12A and 12B show segmentation result images of the dual watershedsegmentation of the present invention versus the standard watershed. Thedual watershed segmentation boundaries are shown as lines superposed onthe original color teeth image in FIG. 12A. The standard watershedregions after segmentation are shown as different gray-valued regions inFIG. 12B. The standard watershed typically oversegments the tooth ofinterest due to color variations within a tooth, as seen in the left andcenter images in FIGS. 12A and 12B. The standard watershed is unable todetect part of the enamel in the image at the right in FIGS. 12A and12B. The dual watershed algorithm of the present invention undersegmentsthe tooth due to the low contrast of the image, but is able to detectthe tooth enamel.

The segmentation results on all the sixty-four test images using thedual watershed algorithm of the present invention are shown in Table 2below. The average segmentation error was 8.1%, including 6.3%oversegmentation errors and 14.4% undersegmentation errors. Note thatthe average segmentation error is below the tolerable (allowable)segmentation error of around 10%. As seen in Table 2, only 14% of theteeth in the test set had a segmentation error of more than 10%. This isquite good, given the variability in color, contrast, and othervariations in the teeth images. As expected, most of theundersegmentation errors were caused by low-contrast images where theboundary between the tooth and background is not clearly visible. FIG.13A shows an example where the tooth is undersegmented because theboundary between the central tooth and the surrounding teeth is of verylow-contrast compared with the glare regions, and the boundary betweenthe enamel and gum regions. An example where tooth segmentation occursdue to the presence of tooth filling at the center of the tooth is shownin FIG. 13B.

TABLE 2 % Teeth % Teeth % Teeth (out of 64( (out of 64( (out of 64( Avg.Avg. Avg. Seg. with >5% with >10% with >20% OverSeg. UnderSeg. Error (%)Seg. Error Seg. Error Seg. Error Error (%) Error (%) Dual 8.1% 21.7% 14%12.5% 6.3% 14.4% Watershed2. Grid Morphine Results

Due to the wide differences in size, shape, and orientations of teeth,the problem of overlaying a grid onto each tooth automatically is anon-trivial task. Since there is no standard rule by which a grid can bedrawn onto an arbitrary tooth, the performance of the automated gridmorphing procedure is incapable of being numerically described.Therefore, an analysis was based on visual examination of thegrid-morphing results. The automatic grid morphing worked robustly andgave intuitively appealing results for all cases where the tooth had oddshapes. FIG. 14A shows a case where the biting surface of the tooth ismissing, thereby creating an awkward-shaped boundary. However, theresultant grid morphed onto this shape, using the template of FIG. 10A,fits this shape very well. The automated grid morphing procedure of thepresent invention also works robustly when the tooth is oriented awayfrom the vertical line, as seen for the tooth in FIG. 14C which isslanted to the left.

3. Plaque Classification Results

A three-class Fisher Linear Discriminant (FLD) linear patternrecognition classifier was used. The mean and covariance matrices ofeach class (plaque, enamel, and glare) were estimated using the red,green, and blue (RGB) values from the sixty-four training image-set.These sample three-by-one mean and three-by-three covariance matriceswere then used to classify the test-set images. The plaqueclassification results were excellent. The three-class confusion matrixis shown in Table 3 below. Only 3.1% of the plaque regions wereincorrectly classified as plaque. This error typically occurs when thecaptured images are blurred due to camera motion. This reduces thecontrast and color of plaque regions. Glare regions are easilydistinguishable from enamel and plaque. FIGS. 15A-15D show examples ofthe classified labeled output image. Note that plaque is detected well(FIGS. 15C and 15D) even when the plaque is present on the enamelsurface that is not well lit. If desired, a red color may indicateplaque, a green color may inficate enamel, and a black color mayindicate glare.

TABLE 3 True class (rows)/ classified as (columns) Plaque Enamel GlarePlaque 96.9%  3.1% 0% Enamel  1.8% 98.2% 0% Glare   0%   0% 100%

The system and method of the present invention provide many advantagesover the related art. For example, the system and method enableautomated evaluation of tooth-brush designs for removing plaque inclinical trial settings. A major component of the system and method isthe development of a new segmentation technique that robustly segmentsobjects in the presence of variable lighting, contrast, color, and noisybackgrounds. This segmentation procedure was shown to work effectivelyon images captured during the course of clinical trials. It was shown tosignificantly outperform the conventional watershed segmentationprocedure. The system and method of the present invention also providesa technique to automatically determine the line correspondences in theBeier-Neely morphing procedure. This is a significant component toachieve automated morphing from one arbitrary shape to another.

The system and method of the present invention also includes utilitiesto dynamically group patients together into examination studies, withdifferent brushes used either in parallel or across different groups ofpatients (in different studies) and software to generate plaque removalefficacy information for different brushes on combinations of teeth, andfor combinations of teeth grid regions. This capability to do automatedplaque analysis on an individual tooth-by-tooth basis with gridinformation data is new and is not believed to have been achieved in thedental community so far.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in the computer-implementedsystem and method for automatically evaluating the efficacy of plaqueremoval and whitening capabilities for different toothbrushes andtoothpastes of the present invention and in construction of this systemand method without departing from the scope or spirit of the invention.As an example, the system and method may include automatic procedures todetermine the image preprocessing parameters (such as the edge-imagenormalization parameters), depending upon each individual image, methodsto handle segmentation when other dyes are used for plaque detection,and methods to handle more angular variations of teeth orientation inthe morphing procedure.

Other embodiments of the invention will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

1. A method for automatically evaluating the efficacy of plaque removaland whitening capabilities for different toothbrushes and toothpastes ona tooth-by-tooth basis in real-time, comprising: determining a centralportion of each tooth with a watershed segmentation algorithm using acenter of each tooth as a seed point; applying a reverse watershedsegmentation algorithm using an external boundary surrounding each toothand an internal portion of each tooth as two seed regions, wherein thepoint at which the two seed regions intersect correspond to the actualtooth boundary; applying a line-based morphing procedure toautomatically overlay a grid on each segmented tooth; and classifyingplaque regions of each tooth.
 2. A method as recited in claim 1, whereinprior to the central portion determination, the method furthercomprises: preprocessing data about each tooth to handle variations inlighting conditions and in the color and contrast of each tooth.
 3. Amethod as recited in claim 2, wherein the preprocessing data comprises:filtering an input color image of the data using edge kernels in eachcolor channel to output edge images; and normalizing each edge imagebased upon a variance of each edge image.
 4. A method as recited inclaim 1, wherein the central portion determination comprises: locating afew pixels on each tooth by determining a cumulative histogram for colorchannels; finding the brightest regions in the color channels; andextracting the largest color region closest to the central region ofeach tooth.
 5. A method as recited in claim 1, wherein the applying areverse watershed segmentation algorithm comprises: growing each toothregion and a background region simultaneously; and labeling as truetooth boundaries the intersections of the tooth and background regions.6. A method as recited in claim 1, wherein the applying a line-basedmorphing procedure comprises: determining a center of gravity of eachsegmented tooth; yielding a line representation of a template tooth andeach tooth with an automatic correspondence between each; and applying aline-based warping rule to produce a grid warped onto each segmentedtooth.
 7. A method as recited in claim 1, wherein classifying plaqueregions of each tooth comprises using a three-class Fisher LinearDiscriminant linear pattern recognition classifier.
 8. Acomputer-implemented method for automatically evaluating the efficacy ofplaque removal and whitening capabilities for different toothbrushes andtoothpastes on a tooth-by-tooth basis in real-time, comprising:determining a central portion of each tooth with a watershedsegmentation algorithm using a center of each tooth as a seed point;applying a reverse watershed segmentation algorithm using an externalboundary surrounding each tooth and an internal portion of each tooth astwo seed regions, wherein the point at which the two seed regionsintersect correspond to the actual tooth boundary; applying a line-basedmorphing procedure to automatically overlay a grid on each segmentedtooth; and classifying plaque regions of each tooth.
 9. Acomputer-implemented method as recited in claim 8, wherein prior to thecentral portion determination, the method further comprises:preprocessing data about each tooth to handle variations in lightingconditions and in the color and contrast of each tooth.
 10. Acomputer-implemented method as recited in claim 9, wherein thepreprocessing data comprises: filtering an input color image of the datausing edge kernels in each color channel to output edge images; andnormalizing each edge image based upon a variance of each edge image.11. A computer-implemented method as recited in claim 8, wherein thecentral portion determination comprises: locating a few pixels on eachtooth by determining a cumulative histogram for color channels; findingthe brightest regions in the color channels; and extracting the largestcolor region closest to the central region of each tooth.
 12. Acomputer-implemented method as recited in claim 8, wherein the applyinga reverse watershed segmentation algorithm comprises: growing each toothregion and a background region simultaneously; and labeling as truetooth boundaries the intersections of the tooth and background regions.13. A computer-implemented method as recited in claim 8, wherein theapplying a line-based morphing procedure comprises: determining a centerof gravity of each segmented tooth; yielding a line representation of atemplate tooth and each tooth with an automatic correspondence betweeneach; and applying a line-based warping rule to produce a grid warpedonto each segmented tooth.
 14. A computer-implemented method as recitedin claim 8, wherein classifying plaque regions of each tooth comprisesusing a three-class Fisher Linear Discriminant linear patternrecognition classifier.
 15. A system for automatically evaluating theefficacy of plaque removal and whitening capabilities for differenttoothbrushes and toothpastes on a tooth-by-tooth basis in real-time,comprising: a memory configured to store instructions; and a processorconfigured to execute the instructions stored in the memory, theinstructions being for: determining a central portion of each tooth witha watershed segmentation algorithm using a center of each tooth as aseed point, applying a reverse watershed segmentation algorithm using anexternal boundary surrounding each tooth and an internal portion of eachtooth as two seed regions, wherein the point at which the two seedregions intersect correspond to the actual tooth boundary, applying aline-based morphing procedure to automatically overlay a grid on eachsegmented tooth, and classifying plaque regions of each tooth.
 16. Asystem as recited in claim 15, wherein prior to the central portiondetermination instructions, the processor executes further instructionsfor: preprocessing data about each tooth to handle variations inlighting conditions and in the color and contrast of each tooth.
 17. Asystem as recited in claim 16, wherein the preprocessing datainstructions comprise: filtering an input color image of the data usingedge kernels in each color channel to output edge images; andnormalizing each edge image based upon a variance of each edge image.18. A system as recited in claim 15, wherein the central portiondetermination instructions comprise: locating a few pixels on each toothby determining a cumulative histogram for color channels; finding thebrightest regions in the color channels; and extracting the largestcolor region closest to the central region of each tooth.
 19. A systemas recited in claim 15, wherein the applying a reverse watershedsegmentation algorithm instructions comprise: growing each tooth regionand a background region simultaneously; and labeling as true toothboundaries the intersections of the tooth and background regions.
 20. Asystem as recited in claim 15, wherein the applying a line-basedmorphing procedure instructions comprise: determining a center ofgravity of each segmented tooth; yielding a line representation of atemplate tooth and each tooth with an automatic correspondence betweeneach; and applying a line-based warping rule to produce a grid warpedonto each segmented tooth.
 21. A system as recited in claim 15, whereinclassifying plaque regions of each tooth instructions comprise using athree-class Fisher Linear Discriminant linear pattern recognitionclassifier.
 22. A computer readable medium stores instructionsexecutable by at least one processor to perform a method forautomatically evaluating the efficacy of plaque removal and whiteningcapabilities for different toothbrushes and toothpastes on atooth-by-tooth basis in real-time, comprising: instructions fordetermining a central portion of each tooth with a watershedsegmentation algorithm using a center of each tooth as a seed point;instructions for applying a reverse watershed segmentation algorithmusing an external boundary surrounding each tooth and an internalportion of each tooth as two seed regions, wherein the point at whichthe two seed regions intersect correspond to the actual tooth boundary;instructions for applying a line-based morphing procedure toautomatically overlay a grid on each segmented tooth; and instructionsfor classifying plaque regions of each tooth.
 23. A computer readablemedium as recited in claim 22, wherein prior to the central portiondetermination instructions, the computer readable medium furthercomprises: instructions for preprocessing data about each tooth tohandle variations in lighting conditions and in the color and contrastof each tooth.
 24. A computer readable medium as recited in claim 23,wherein the preprocessing data instructions comprise: instructions forfiltering an input color image of the data using edge kernels in eachcolor channel to output edge images; and instructions for normalizingeach edge image based upon a variance of each edge image.
 25. A computerreadable medium as recited in claim 22, wherein the central portiondetermination instructions comprise: instructions for locating a fewpixels on each tooth by determining a cumulative histogram for colorchannels; instructions for finding the brightest regions in the colorchannels; and instructions for extracting the largest color regionclosest to the central region of each tooth.
 26. A computer readablemedium as recited in claim 22, wherein the applying a reverse watershedsegmentation algorithm instructions comprise: instructions for growingeach tooth region and a background region simultaneously; andinstructions for labeling as true tooth boundaries the intersections ofthe tooth and background regions.
 27. A computer readable medium asrecited in claim 22, wherein the applying a line-based morphingprocedure instructions comprise: instructions for determining a centerof gravity of each segmented tooth; instructions for yielding a linerepresentation of a template tooth and each tooth with an automaticcorrespondence between each; and instructions for applying a line-basedwarping rule to produce a grid warped onto each segmented tooth.
 28. Acomputer readable medium as recited in claim 22, wherein classifyingplaque regions of each tooth instructions comprise instructions forusing a three-class Fisher Linear Discriminant linear patternrecognition classifier.